Augmented Reality Content Creation

ABSTRACT

A system and method for generating a virtual content to a physical object is described. The system accesses a video that was captured by a camera. The video depicts a user performing a physical operation in relation to a physical object. The system determines metadata identifying positions of the camera relative to the physical object during the capturing of the video, identifies feature points of the physical object in the video, and retrieves a virtual object based on the feature points of the physical object. The system generates virtual content based on the video, the metadata, and the virtual object, and stores the object identifier and the virtual content relationally in a data storage that stores multiple sets of virtual content associated with multiple physical objects and their corresponding object identifiers.

PRIORITY APPLICATION

This application is a continuation of co-pending U.S. application Ser.No. 17/483,448, filed Sep. 23, 2021, which is a continuation of U.S.application Ser. No. 15/043,959, now U.S. Pat. No. 11,150,482, whichclaims priority to U.S. Provisional Application Ser. No. 62/120,808,filed Feb. 25, 2015, the disclosures of which are incorporated herein intheir entirety by reference.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to the processingof data. Specifically, the present disclosure addresses systems andmethods for generating augmented reality content using a head mounteddevice.

BACKGROUND

Augmented reality (AR) is a live, direct or indirect view of a physical,real-world environment whose elements are augmented bycomputer-generated sensory input such as sound, video, graphics, orglobal positioning system (GPS) data. With the help of advanced ARtechnology (e.g., adding computer vision and object recognition) theinformation about the surrounding real world of the user becomesinteractive. Device-generated (e.g., artificial) information about theenvironment and its objects can be overlaid on a display through which auser views objects in the real world.

The user can create AR content associated with physical objects using anAR content software tool and the AR content may then be stored on aserver. In certain circumstances, the AR content may be provided to anAR device (or devices) when the AR device recognizes a correspondingphysical object.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings.

FIG. 1 is a block diagram illustrating an example of a networkenvironment suitable for creating virtual content for a physical object,according to some example embodiments.

FIG. 2 is a block diagram illustrating an example embodiment of modules(e.g., components) of a viewing device.

FIG. 3 is a block diagram illustrating example modules of a virtualcontent creation module.

FIG. 4 is a block diagram illustrating example modules of a virtualcontent training module.

FIG. 5 is a block diagram illustrating example modules of a virtualcontent rendering module.

FIG. 6 is a block diagram illustrating example modules executing on oneor more processors of a server.

FIG. 7 is an interaction diagram illustrating interactions between aviewing device for creating virtual content and a server.

FIG. 8 is an interaction diagram illustrating interactions between aviewing device for accessing virtual content and a server.

FIG. 9 is a flowchart illustrating an example operation of authoringvirtual content from a viewing device for an augmented realityapplication.

FIG. 10 is a flowchart illustrating an example operation of retrieving avirtual content at a viewing device.

FIG. 11 is a diagram illustrating an example operation for generatingvirtual content related to a physical object for an augmented realityapplication at a mobile device.

FIG. 12 is a block diagram illustrating components of a machine,according to some example embodiments, able to read instructions from amachine-readable medium and perform any one or more of the methodologiesdiscussed herein.

FIG. 13 is a block diagram illustrating a mobile device, according to anexample embodiment.

FIG. 14 is a diagram illustrating a front view of a head mounted device.

FIG. 15 is a diagram illustrating a side view of a head mounted device.

DETAILED DESCRIPTION

Example methods and systems are directed to creation of virtual contentfrom a head mounted device. Examples merely typify possible variations.Unless explicitly stated otherwise, components and functions areoptional and may be combined or subdivided, and operations may vary insequence or be combined or subdivided. In the following description, forpurposes of explanation, numerous specific details are set forth toprovide a thorough understanding of example embodiments. It will beevident to one skilled in the art, however, that the present subjectmatter may be practiced without these specific details.

Augmented reality (AR) applications allow a user to experience ARcontent, such as in the form of a virtual object displayed in a viewingdevice. The viewing device may include a display, a camera, and an ARapplication implemented by a hardware processor. The virtual object maybe associated with a physical object perceived with the camera of theviewing device. The physical object may include a visual reference(e.g., QR code on a surface of the physical object or feature points ofthe physical object) that the AR application can identify. The ARapplication identifies the visual reference from an image or video frameof the physical object and retrieves the virtual object associated withthe visual reference or captured image of the physical object. Theviewing device may locally render the virtual object based on a positionof the viewing device relative to the physical object. The virtualobject may include a three-dimensional virtual object or atwo-dimensional virtual object. For example, the three-dimensionalvirtual object may include a three-dimensional model of a circuitswitch. The two-dimensional virtual object may include a two-dimensionalview of a dialog box, menu, or statistics information related to thephysical object. AR content may be alternatively referred to as virtualcontent.

In an example embodiment, the viewing device includes a head mounteddevice (HMD), such as a helmet or eyewear, that can be worn by a user.The HMD may also include transparent lenses through which the user canview AR content. The transparent lenses may include a transparentdisplay to generate a visualization of the AR content in a light ofsight between the eyes of the user and the physical object being watchedby the user. The HMD can be a standalone device that can be used tocreate virtual content without using aclient device (e.g., laptop ordesktop). Accordingly, in an example embodiment, the actual HMD that canbe used for presenting AR content to a user, at least partially, createsthe virtual content (which may, optionally, be displayed by other HMDs).Thus, the wearer of the HMD may determine (other than another thirdparty) virtual content to be associated with a physical object. Thewearer of the HMD may also be referred to as the user of the HMD.

Virtual content includes multimedia content that can be played in theHMD. For example, multimedia content includes video, audio, andgraphical animation content. Visual content can be displayed in thedisplay of the HMD. Audio content can be played through built-inspeakers in the HMD.

In an example embodiment, the HMD associates the virtual content withidentifiers of the physical object and tracking data related to thephysical object. Example identifiers include a QR code or other visualidentifying reference, such as feature points, that can be visuallyscanned using the camera of the HMD. Examples of tracking data includewireless identification signals broadcasted from the physical object(e.g., Bluetooth, beacon signals). Other tracking data include locationdata of the physical object as determined by a GPS location of the HMDin proximity to the physical object, an orientation of the HMD, and adistance between the location of the HMD and the location of thephysical object. Sensors included in the physical object may alsogenerate tracking data (e.g., GPS data, status data). For example, thephysical object may include a switch with a sensor that broadcasts awireless switch status signal indicating the position of the switch(e.g., toggled position). Therefore, the HMD may use the identifiers ofthe physical object to identify the physical object and retrieve thecorresponding virtual content associated with the physical object. TheHMD may further use the tracking data of the physical object to track astatus of the physical object and to track operations performed on thephysical object by the user of the HMD by use of sensors in the HMD andin the physical object. The operations may include, for example, turningon a switch, physically moving a component of the physical object, suchas opening a circuit panel.

An example way to capture the operations performed on the physicalobject is by recording a video of the wearer of the HMD interacting withthe physical object from the point of view of the user wearing the HMD.For example, a camera in the HMD records the wearer of the HMD repairingthe physical object. The video may be divided into different video clipsthat are associated with specific positions or orientations of the HMDrelative to the physical object. Video or parts thereof may then bestored at a server that can provide (e.g., via a wireless network) thevideo or part thereof to another HMD. Each video may be associated withmetadata that include information related to positions (e.g., 2 feetaway from a machine) and orientations (e.g., HMD looking at the front ofa machine) of the HMD relative to the physical object. For example, avideo clip showing an authoring user operating components on a machine(e.g., turning dials, flipping switches, attaching or detaching circuitcomponents) may be displayed in the HMD of another user or subsequentuser when the HMD of that other user is at a location of the machine andwithin a position and orientation associated with the video clip. Onevideo clip may be associated with a specific orientation and position ofthe HMD relative to the machine. In other embodiments, several videoclips may be associated with a specific orientation and position of theHMD relative to the machine. In that situation, the HMD identifies andplays the relevant video clip based on a condition of the machine (e.g.,overheated, normal operation, specific error code) as determined bysensors in the HMD or in the machine. Therefore, a video clip related tohow to fix an overheated machine may be displayed in the HMD when thewearer of the HMD is oriented towards the front of a machine, and thecondition of the machine shows that it is overheated. Similarly, adifferent video clip related to how to address a specific error code maybe displayed in the HMD when the HMD is in the same previous locationand orientation, and the condition of the machine is associated with thespecific error code. Accordingly, AR content authored by one user of anHMD may then be provided to and displayed in other HMDs. The wearer ofthe HMD authoring the content is also referred to as an authoring user.

Another example of a way to capture the operations performed on thephysical object is by tracking relative locations of identifying featurepoints in an image or a video frame of the physical object and recordingin the changes in location of the identifying feature points. Theidentifying feature points may be corner points of a printed square,corner points of physical objects, or points of high contrast ratio. Thefeature points, themselves, do not change when viewed from anyobservation angle. The HMD tracks relative movement of the featurepoints to map movement of the physical object.

For example, a user wearing an HMD opens a panel or enclosure and viewsa circuit board through the lenses of the HMD. The HMD may firstidentify the panel based on predefined identifying feature points andsensor data fromthe HMD and the circuit board (e.g., corner points ofthe panel, GPS location of the HMD, wireless signal broadcasting an IDfrom the circuit board and the panel). The HMD may then track how thepanel or a door of an enclosure has moved based on changes of the cornerpoints of the panel. For example, the HMD may track and record spatialgeometric points (e.g., corner points) on the panel. The opening of thepanel is tracked and recorded in the form of changes to the spatialgeometric points. The HMD can use the changes to the spatial geometricpoints to generate virtual content. For example, the HMD generates avirtual arrow to indicate a direction in which to open the panel basedon the changes to the location of the corner points of the panelperceived by the HMD and sensor data from sensors on the panel.

The AR content may be displayed in a HMD when the HMD is positioned in aline of sight of the wearer of the HMD and the HMD is directed to thephysical object at the same (or similar point) in a three dimensionalspace as the HMD that originated or initially generated the AR content.For example, when another or second HMD views the same physical object,the second HMD retrieves the originally authored virtual content. Theoriginal authored virtual content may then be communicated to the secondHMD which may then show a video clip or an animation of a virtual panelbeing opened just like when the authoring user opened the panel. Thesecond HMD may trigger a display of the animation or video clip based onpreset conditions such as when the user approaches and views the panelfrom a specific orientation and position at a particular time or basedon specific conditions. An example of a specific condition may be whenthe user is authorized to open the panel based on credentials of theuser of the HMD. Another example of a specific condition may be when amalfunction of equipment has occurred and the panel provides access toreboot, restart, or otherwise provide corrective maintenance.

It will be noted that the virtual content that is, directly orindirectly (e.g., by producing an animation), authored using a first HMDand subsequently communicated to one or more other HMDs, may includerecorded media, a two-dimensional or three-dimensional model animation,and text. A library of virtual content may be created wherein eachcontent item may be associated with a different physical object (e.g., aparticular machine in a power plant, manufacturing facility, etc.). Forexample, the HMD may record the user performing steps to repair amachine. The recorded steps may be in the form of a media (e.g., videoclip) or tracking data (e.g., tracking the changes in the relativelocation of the identifying feature points). A graphics animation (e.g.,virtual arrow showing a direction in which to flip a switch) may begenerated based on the tracking data. The virtual content may includethe media or the graphics animation. The virtual content is uploaded toa server and stored and tagged or associated with repairing theparticular machine. Accordingly, another user of an HMD encountering thesame or similar machine can retrieve and visualize the repair steps fromthe virtual content using his or her HMD that is communicatively coupled(e.g., via one or more networks) to a server or an AR contentrepository. The user or any other subsequent user may, using his or herHMD, see video clips associated with each repair step or can seeanimation of virtual objects (e.g., virtual pieces or arrows, audioinstructions, wiring diagrams, etc.) associated with each repair step.

In an example embodiment, one or more servers receive the virtualcontent (e.g., captured video footage, graphics animation, tracking databased on location changes to the identifying feature points of aparticular machine) generated by the authoring user of the authoringHMD, and associate or tag the virtual content to an identifier of amachine. The identifier uniquely identifies the machine using, forexample, a location of the machine or a serial number of the machine.One or more servers store the virtual content, the correspondingidentifier of the machine, and corresponding parameters or conditionsassociated with the virtual content. Examples of parameters orconditions may include a temperature range of a machine located at aspecific location, an operating status of the machine, and credentialsof the wearer of the HMD. When a location, status, orientation, orposition of another HMD meets the parameters or conditions associatedwith the virtual content, the servers provide the corresponding virtualcontent to the HMD. Thus, when or where the virtual content (e.g., videoor animations) is communicated to the HMD may be dependent on the wearerof the HMD being in proximity to a physical object (e.g., an industrialmachine, a machine, an apparatus) associated with the virtual content,the nature of the problem or malfunction of the physical object orapparatus associated with the virtual content, the operational state ofthe physical object or apparatus associated with the virtual content.For example, the server provides virtual content (e.g., video footage)demonstrating how to operate an industrial machine to the HMD when theHMD is located in proximity to the industrial machine. In anotherexample, the server detects the nature of the malfunction of theindustrial machine based an error code output from the industrialmachine. The server retrieves the relevant video footage associated withthe error code and provides the relevant video footage to the HMDlocated in proximity to the industrial machine. In yet another example,the server determines the technical level (e.g., apprentice or advanced)of the wearer of the HMD based on an identification or authenticationcredentials of the wearer of the HMD. The wearer of the HMD may beauthenticated, for example, using biometrics authentication methods(e.g., iris scan of the eyes of the wearer of the HMD). The serverretrieves the relevant video footage associated with the correspondingtechnical level of the wearer of the HMD. Therefore, a video footageincluding advanced technical descriptions may be provided to a wearer ofthe HMD with an advanced technical expertise as determined by theauthentication credentials of the wearer of the HMD.

In one example embodiment, the AR content may be associated to acorresponding geographical location, a process a user is performing(e.g., fixing an engine), ambient information (e.g., humidity andtemperature), a tagged location or actions by other users, and specifictype/brand/model of machines. The AR content may be accessed fromanother HMD device or a client device that can further edit the ARcontent by modifying, supplementing, deleting, changing portions of theAR content. For example, the client device can modify a color or shapeof a three-dimensional model of a physical object in the AR content. Theclient device can access the AR content from the server using aweb-based application to edit and adjust the size, color, or shape of avirtual arrow in the AR content. The client device can also use theweb-based application to trim a duration of a video footage in the ARcontent, combine different video footages into one video footage in theAR content, and associate the video footage with another physicalobject. The client device can change parameters associated with the ARcontent and the physical object. For example, the client device canchange the temperature threshold to trigger the AR content for aparticular industrial machine.

An authoring user of a HMD may generate custom content or may selectfrom a library or repository of AR content stored on the HMD orretrievable from a remote server via a network. The library may includethree-dimensional models of virtual content of similar physical objects(e.g., machines having similar shape, machines located on the secondfloor of a factory, or machines located within a user defined area).Accordingly, the authoring user of the HMD selects a three-dimensionalmodel of virtual content of a physical object similar to the physicalobject in proximity to the HMD. The authoring user of the HMD may selectthe model using voice command or other user interface means whilenavigating a menu displayed in the display of the HMD. Furthermore, theauthorizing user of the HMD may create interactive content or a staticmenu of information identifying the physical object (e.g., apparatus)being viewed or pointed to by the HMD. In another example embodiment,the virtual content may include interactive virtual functions associatedwith functions of the physical object (e.g., a virtual red button whenactivated stops the particular machine). The virtual content may alsodisplay dynamic information, such as status update (e.g., virtual greenlight to indicate that the machine is operating as expected).

In an example embodiment, the HMD can detect, generate, and identifyidentifiers such as feature points (e.g., unique spatial geometricattributes) of a physical object being viewed or pointed to by the HMD.The HMD captures an image or a video frame of the physical object. TheHMD determines identifiers in the image or video frame of the physicalobject and identifies the physical object based on the identifiers andtracking data (e.g., GPS location of the physical object) related to thephysical object. The HMD may then retrieve the virtual contentassociated with the physical object. The HMD may include a screen todisplay the retrieved virtual content in relation to an image of thephysical object captured by a camera and displayed on a display screenof the HMD. In another example embodiment, the HMD includes atransparent display that can display the retrieved virtual content inrelation to a position and an orientation of the HMD relative to thephysical object, as viewed by a user through the transparent display.Sensors in the HMD may be used to determine the location and theorientation of the HMD relative to the physical object in the physicalenvironment of the HMD. The transparent display may be positioned in theHMD such that the user can view the physical object through thetransparent display. The location may include a geographic locationdetermined based on wireless data generated by the HMD or triangulatedfrom the predefined references of the physical environment, a GPS deviceor any other location identifier. The orientation may be determinedbased on gyroscope data from an inertial measurement unit (IMU) sensorsprovided in the HMD or externally determined using a three-dimensionalcamera sensor.

In one example embodiment, the HMD communicates with a server that has astorage device for storing a database or library of virtual contentgenerated at the HMD or from other HMDs, corresponding identifiers ofthe physical objects, and parameters or tracking data related to thephysical objects. Another HMD may send to the server identifierscaptured from the physical object, tracking data, and other sensor data(e.g., location, ambient information). The server may then identify thephysical object based on the identifiers of the physical object (and,optionally, the tracking data) and query the database. The database mayinclude a plurality of reference identifiers each corresponding to aphysical object, or features thereof, that identify virtual content. Inthis manner, virtual content corresponding to the identifier receivedfrom the HMD may be identified. The server may then retrieve the virtualcontent (e.g., a video footage or a three-dimensional virtual objectanimation) associated with the identified physical object and the othersensor data and communicate the virtual content to the HMD. It will beappreciated that at least part of the database may reside locally on theHMD, may be remotely provided, or any combination thereof. Furthermore,each HMD can store a database that is customized for a particular userso that most of the information relevant to the particular user isalready available locally. For example, virtual content related tomanufacturing equipment commonly operated by a worker may be locallystored in the HMD of the worker. Virtual content related to specialmanufacturing equipment which the worker has privilege access may belocally stored in the HMD of that worker.

In another example embodiment, the server receives an image of thephysical object and tracking data from a HMD. The server may analyze theimage, for example, to generate feature points from the image of thephysical object. The server may then identify a physical object based onthe generated feature points of the physical object and tracking dataafter querying the database of reference identifiers of physical objectsand tracking data related to physical objects. Finally, the server mayretrieve the virtual content associated with the identified physicalobject and communicate it to the HMD for presentation to the user.

A non-transitory machine-readable storage device may store a set ofinstructions that, when executed by at least one processor, causes theat least one processor to perform any one or more of the methodologiesdescribed within the present disclosure.

FIG. 1 is a network diagram illustrating a network environment 100suitable for creating virtual content for a physical object, accordingto some example embodiments. The network environment 100 includes aviewing device 101 and a server 110, communicatively coupled to eachother via a network 108. The viewing device 101 and the server 110 mayeach be implemented in a computer system, in whole or in part, asdescribed below with respect to FIGS. 2 and 6 .

The server 110 may be part of a network-based system. For example, thenetwork-based system may be or include a cloud-based server system thatprovides virtual content such, as three-dimensional models or videofootage, to the viewing device 101.

A user 102 may utilize the viewing device 101 to view a physical object(e.g., factory machine) in a local real world environment such as at afactory 103. The user 102 may be a human user (e.g., a human being), amachine user (e.g., a computer configured by a software program tointeract with the device 101), or any suitable combination thereof(e.g., a human assisted by a machine or a machine supervised by ahuman). The user 102 is not part of the network environment 100, but isassociated with the viewing device 101 and may be a user 102 of theviewing device 101. For example, the viewing device 101 may be a HMD(e.g., wearable computing device with a display that is worn on a headof the user 102). The computing device may also be worn on other partsof the body (e.g., arms or hip) of the user 102. In one exampleembodiment, the display may be transparent or semi-transparent such asin lenses of wearable computing glasses.

The user 102 may use the viewing device 101 to author virtual contentusing an AR application in the viewing device 101. The AR applicationmay also provide the user 102 with AR content triggered by a view of thephysical object and other data points related to the physical object orto the viewing device 101. The physical object may include atwo-dimensional physical object (e.g., a picture) or three-dimensionalphysical object (e.g., a factory machine 114). Examples of data pointsrelated to the physical object may include location information (e.g.,first floor of a factory facility, GPS coordinates) or spatial geometricreferences (e.g., perceived corners of walls or furniture, uniqueidentifiers or markers) in the real world physical environment. Forexample, the user 102 may direct a camera of the viewing device 101 tocapture an image of the factory machine 114. The viewing device 101identifies feature points (e.g., edges, corners, surface of the machine,unique spatial geometric patterns) in the picture or video frame of thefactory machine 114 to identify the factory machine. The viewing device101 accesses a local library (e.g., local context recognition dataset orany other previously stored dataset of the AR application) of theviewing device 101 to retrieve a virtual object corresponding to thefeature points. The local library may also include models of virtualobjects associated with feature points of real-world physical objects orreferences. In another example embodiment, the viewing device 101identifies tracking data related to the factory machine 114. Examples oftracking data include GPS location of the factory machine 114, uniquewireless identifier, operating status, and distance from the userviewing device 101 to the factory machine 114.

In an example embodiment, the viewing device 101 enables the user 102 tocreate virtual content for the factory machine 114 from the viewingdevice 101 by recording a video the user 102 operating on the factorymachine 114. For example, the viewing device 101 may record a videofootage of the user 102 fixing the factory machine 114. The user 102 mayuse different types of user interfaces to start and stop the videorecording. Examples of user interfaces include a voice command, agesture command by waving the user's hand in a predefined pattern, and atouch sensitive area in the viewing device. The video footage may betagged with metadata identifying a position of the viewing device 101relative to the factory machine 114. For example, the metadata mayinclude information identifying the distance between the viewing deviceMI and orientation of the viewing device 101 relative to the factorymachine 114. The relative distance and orientation of the viewing device101 may be measured with internal sensors (e.g., Inertial MeasurementUnit sensors, depth sensors) in the viewing device 101, external sensors(e.g., depth sensor cameras in the factory 103 or on the factory machine114), or a combination thereof. The video footage and correspondingmetadata is stored in the viewing device 101 and later can be offloadedto the server 110.

The viewing device 101 may also monitor and track changes in thelocation of feature points of a component of the factory machine 114relative to feature points of other components of the factory machine114 in the video footage. For example, feature points on a circuit panelmay move relative to other feature points of the factory machine 114.Consequently, the viewing device 101 can determine and identify that acomponent has moved in a particular direction (e.g., a switch has beenflipped in a direction) based on the relative motion of the featurepoints. The viewing device 101 generates a virtual object based on therelative changes of the feature points in the factory machine 114. Forexample, the virtual object may include an animation of a virtualthree-dimensional arrow indicating a direction in which the flip istoggled. The virtual object is stored in the viewing device 101 andlater can be offloaded to the server 110. Therefore, the viewing device101 generates a video footage, a virtual object, or a combinationthereof. The video footage and virtual object may be referred to asvirtual content.

In another example embodiment, the server 110 receives the virtualcontent authored and created at the viewing device 101. The virtualcontent may also be available to other users of other viewing devices.Consequently, a second user of a second viewing device can access andretrieve the virtual content authored by a first user when the locationof the second viewing device worn by the second user is in proximity tothe factory machine 114. The location and orientation of the secondviewing device may be detected with internal sensors in the secondviewing device, or with external tracking sensors 112. In anotherexample embodiment, the server 110 provides the virtual content aboutthe factory machine 114 to other viewing devices when the other viewingdevices are within a predefined distance range of the factory machine114 (e.g., 2 to 5 feet distance between a viewing device and the factorymachine 114) and within a predefined position and orientation rangerelated to the factory machine 114 (e.g., a viewing device is in frontof the factory machine 114 and is level—not looking up or down). Theviewing device 101 may define the distance range, the position andorientation range, and associate the predefined ranges with the virtualcontent.

Furthermore, the AR application in the viewing device 101 and the server110 can subsequently modify the originally authored virtual content andgenerate additional content (e.g., additional virtual content)corresponding to the factory machine 114. If the factory machine 114 isnot recognized locally at the viewing device 101, the viewing device 101downloads additional information (e.g., a three-dimensional model of thefactory machine based on detected feature points) corresponding to thefactory machine 114, from a database of the server 110 over the network108.

In another example embodiment, the tracking sensors 112 may be used totrack the location and orientation of the viewing device 101 externallywithout having to rely on the sensors internal to the viewing device101. The tracking sensors 112 may include optical sensors (e.g.,depth-enabled 3D camera), wireless sensors (Bluetooth, Wi-Fi), GPSsensor, and audio sensor to determine the location of the user 102wearing the viewing device 101, the distance of the user 102 to thetracking sensors 112 in the physical environment (e.g., sensors placedin corners of a venue or a room), the orientation of the viewing device101 based on a visual marker or indicator on the viewing device 101.

In another example embodiment, data from the tracking sensors 112 andinternal sensors in the viewing device 101 may be used for analyticsdata processing at the server 110 for analysis on usage and how the user102 interacts with the factory machine 114. Examples of analytics datamay include the locations (e.g., feature points) on the physical objectthe user 102 has viewed, how long the user 102 has looked at eachlocation on the physical object, which video clips or footage have beenviewed, and how long the user 102 has watched the video clip.

Any of the machines, databases, or devices shown in FIG. 1 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform oneor more of the functions described herein for that machine, database, ordevice. For example, a computer system able to implement any one or moreof the methodologies described herein is discussed below with respect toFIGS. 9, 10 . As used herein, a “database” is a data storage resourceand may store data structured as a text file, a table, a spreadsheet, arelational database (e.g., an object-relational database), a triplestore, a hierarchical data store, or any suitable combination thereof.Moreover, any two or more of the machines, databases, or devicesillustrated in FIG. 1 may be combined into a single machine, and thefunctions described herein for any single machine, database, or devicemay be subdivided among multiple machines, databases, or devices.

The network 108 may be any network that enables communication between oramong machines (e.g., the server 110), databases, and devices (e.g.,viewing device 101). Accordingly, the network 108 may be a wirednetwork, a wireless network (e.g., a mobile or cellular network), or anysuitable combination thereof. The network 108 may include one or moreportions that constitute a private network, a public network (e.g., theInternet), or any suitable combination thereof.

FIG. 2 is a block diagram illustrating an example embodiment of modules(e.g., components) of the viewing device 101. The viewing device 101 mayinclude sensors 202, a display 204, a processor 206, and a storagedevice 208. The viewing device 101 may be a wearable computing devicesuch as a smart helmet. The viewing device 101 may also be referred toas a HMD.

The sensors 202 may include, for example, a proximity or location sensor(e.g., Near Field Communication, GPS, Bluetooth, Wi-Fi), an opticalsensor (e.g., camera, depth sensors), an orientation sensor (e.g.,gyroscope, Inertial Measurement Unit), an audio sensor (e.g., amicrophone), or any suitable combination thereof. For example, thesensors 202 may include rear facing cameras and front facing cameras inthe viewing device 101. It is noted that the sensors 202 describedherein are for illustration purposes and that the sensors 202 are notlimited to the ones described. The sensors 202 may be used to generateinternal tracking data of the viewing device 101 to determine what theviewing device 101 is capturing or looking at in the real physicalworld.

The display 204 may include, for example, a transparent or a see-throughdisplay so that the user can see through the display 204 (e.g., Head-UpDisplay). The display 204 includes a screen or monitor configured todisplay images generated by the processor 206. The display 204 may beincluded and formed in lenses of the HMD. The lenses may be positionedadjacent to the eyes of the wearer of the HMD so that the lenses are inthe light of sight of the wearer.

The viewing device 101 may include an augmented reality (AR) application212, executing on the processor 206, for creating and generating virtualcontent. The virtual content may be displayed when the viewing device101 is within predefined conditions associated with the virtual contentand the corresponding physical object. An example of predefinedconditions may be displaying video clip A when in room B between thehours of 10 μm and 6 am, when temperature of machine C is above xdegrees. In one example embodiment, the AR application 212 may include avirtual content creation module 214, a virtual content training module216, and a virtual content access module 218.

The virtual content creation module 214 enables the user of the viewingdevice 101 to create virtual content related to a physical object. Forexample, the virtual content may include information related to thephysical object (e.g., video or audio description of how to operate thephysical object, animation showing virtual arrows around the physicalobject with a corresponding audio description). The user may create thisvirtual content using the viewing device 101 without having to use aclient device such as a desktop or laptop computer. FIG. 3 is a blockdiagram illustrating example modules of the virtual content creationmodule 214.

The virtual content creation module 214 may include a standard contentmodule 302 and a custom content module 304. The standard content module302 may include a standard template related to the type of physicalobject (e.g., a drilling machine, a heating machine). For example, thestandard template may include a factory name, a machine name, authorizedoperators of the machine, and a description of an operation beingdemonstrated on the machine (e.g., how to repair the machine, how tooperate the machine). The standard template may be associated withphysical objects having similar characteristics, such as location,operators, and type of machine. Consequently, machines located within asame plant may share the same content template.

The user of the viewing device 101 may then select a standard templatebased on the type of machine (e.g., drilling or heating machine) or thetype of information that needs to be provided (e.g., maintenancecleaning, regular operation, diagnostic operation, or repair). Selectionof the standard template may be accomplished via a variety of inputssuch as audio commands, touch input on a surface of the viewing device101, predefined visual gestures detected by internal sensors 202,external sensors 112, or a combination thereof. The standard templatemay be displayed in the display 204 of the viewing device 101. In oneexample embodiment, upon selection of the standard template, the virtualcontent creation module 214 generates audio and/or visual cues for theuser to “fill out” the standard template. For example, the viewingdevice 101 may generate a series of audio prompts for the user toprovide content. An example of an audio prompt may include “Please namethe machine you are looking at.” The user may reply by saying the nameof the machine (e.g., “floor drilling machine B”) and speech recognitionmay then be used to identify the user's vocal response. The next audioprompt may be “repair or operation?” In response to an “operation” replyfrom the user, the virtual content creation module 214 may startrecording a video, using a camera of the viewing device 101, of the useroperating the machine. In another example, in response to the“operation” vocal reply from the user, the virtual content creationmodule 214 tracks feature points of the machine (or any apparatus) todetermine components of the machine that are being or have been movedduring a normal operation of the machine. For example, the virtualcontent creation module 214 may track a flipping of a switch by trackingpredefined feature points on the physical switch. In an exampleembodiment, the virtual content creation module 214 automaticallygenerates an animation of a three-dimensional model of the switch behindflipped corresponding to the tracked predefined points. In anotherexample embodiment, the virtual content creation module 214 generates athree-dimensional model of an arrow being animated to show the switchdirection. In yet another example embodiment, the location changes ofthe feature points may be tracked and communicated to a remote serverthat generates an animation of a three-dimensional virtual object basedon the changes in the feature points. In yet another example embodiment,the video of the user flipping the switch may be communicated to aremote server that generates the animation based on an analysis of thevideo.

The custom content module 304 allows the user to customize a template byintroducing different categories of content from the standard template.Accordingly, in a custom mode, the user does not follow particularprompts predefined in a standard template. For example, instead ofproviding a name of a machine, a description, a location, and ademonstration of an operation of the machine, the user can just providea video clip in the custom template (as opposed to a predefined templatethat dictates information to be provided by the user). For example, thecustom template may include the video clip and corresponding predefinedconditions. An example of a predefined condition is when the temperatureof a factory machine exceeds a temperature threshold or when the viewingdevice 101 is positioned in a particular location and orientation withrespect to the factory machine 114. Selection of the custom template maybe similarly accomplished via audio commands, touch input on a surfaceof the viewing device 101, predefined visual gestures detected byinternal sensors 202, external sensors 112, or a combination thereof. Inone example embodiment, upon selection of the custom template, thevirtual content creation module 214 generates audio and/or visual cuesfor the user to define the custom template. For example, the viewingdevice 101 may generate a series of audio prompts for the user toprovide categories for the custom template. An example of an audioprompt may include “Please define a first category for your customcontent.” The user may reply with “machine ID” and speech recognitionmay then be used to identify the user's vocal response. The next audioprompt may be “Please define a second category for your custom content.”The user may reply with “video demonstration of a repair” and speechrecognition may then be used to identify the user's vocal response. The“video demonstration” may be keyword that triggers the viewing device101 to prompt: “please say ‘record’ when you are ready and ‘stop’ whenyou are finished.” In response to a “record” vocal reply from the user,the virtual content creation module 214 may start recording a video ofthe user repairing the machine. The user can further customize thecustom template by specifying a predefined conditions category underwhich the video clip is to be played. Examples of predefined conditionsinclude a location and position of the viewing device 101 while theviewing device 101 is recording the user repairing the machine.Therefore, a video clip showing a user fixing a front panel of factorymachine may be triggered when the viewing device 101 is facing the frontpanel of the factory machine 114. In the same manner, when the user onthe left side of the factory machine 114 looking down, a video clipcorresponding to the location of the user and the orientation isretrieved and displayed.

Referring back to FIG. 2 , the virtual content training module 216allows the user of the viewing device 101 to train the augmented realityapplication 212 to associate the virtual content, created with thevirtual content creation module 214, with the corresponding physicalobject viewed by the viewing device 101. FIG. 4 is a block diagramillustrating example modules of the virtual content training module 216.The virtual content training module 216 is shown by way of example toinclude a physical object identifier module 402 and a training module404.

The physical object identifier module 402 may detect, generate, andidentify identifiers such as feature points of the physical object beingviewed using the viewing device 101. Based on the identifiers, thephysical object identifier module 402 may identify a physical object.However, because machines may resemble one another on a factory floor,the physical object identifier module 402 may use tracking data (e.g.,unique feature points, a quick response (QR) code displayed on amachine, a serial number, or the like) to further assist in identifyingthe unique physical object. After the physical object identifier module402 determines the identifiers of the physical object, the physicalobject identifier module 402 accesses a local library of physicalobjects information and corresponding identifiers in the storage device208 (or in a library stored in a remote server or on another viewingdevice). The physical object identifier module 402 compares theidentifiers of the physical object to identifiers in the library in thestorage device 208 and retrieves an identification of a correspondingphysical object. For example, the physical object identifier module 402identifies the QR code on the factory machine 114, compares the QR codeto a library of QR codes in the storage device 208, and retrieves anidentification of the factory machine 114. The identification of thefactory machine 114 may include information that uniquely identifies thefactory machine 114. Examples of information include a serial number, atag number, location information or coordinate, or any unique identifierassociated with the factory machine 114. Other information about thefactory machine 114 may be included in the identification information.For example, the make, model, operating status, type of factory machine,related factory machines, factory machines connected to the factorymachine 114 may be included in the identification information. Inanother example embodiment, the physical object identifier module 402displays the identification of the factory machine and requests aconfirmation to the wearer of the viewing device 101 to confirm theidentification of the factory machine 114.

In another example embodiment, the physical object identifier module 402determines a geographic location of the user wearing the viewing device101 using sensors 202 internal to the viewing device 101 or sensors 112external to the viewing device 101, or a combination thereof. Thephysical object identifier module 402 accesses a library of physicalobjects information and corresponding geographic locations in thestorage device 208 (or in a library stored in a remote server or onanother viewing device). The physical object identifier module 420compares the determined geographic location of the viewing device 101with the geographic locations in the library in the storage device 208and retrieves an identification of the corresponding physical object.For example, the physical object identifier module 402 identifies thatthe viewing device 101 is located at a specific GPS coordinate, comparesthe specific GPS coordinate to a library of GPS coordinates in thestorage device 208, and retrieves a corresponding identification of thefactory machine 114.

The training module 404 trains the augmented reality application 212 toassociate an identification of the factory machine 114 as determined bythe physical object identifier module 402 with the virtual contentgenerated with the virtual content creation module 214. For example, thetraining module 404 creates an association link in a library databasebetween virtual content A and identifier A of machine A. In otherexample embodiments, the training module 404 accesses and retrievesvirtual content associated a geographic location of the factory machine114 based on a geographic location of the viewing device 101. Thetraining module 404 also retrieves the identification of the factorymachine 114 based on the geographic location of the viewing device 101.Subsequently, the training module 404 associates the virtual contentbased on the geographic location of the factory machine 114 with theidentification of the factory machine 114 based on the geographiclocation of the viewing device 101 when the virtual content was created.

In other examples, the training module 404 determines that virtualcontent is associated with a specific type of machine (e.g., a drill).The training module 404 determines from the identification informationof the factory machine 114 as determined by the physical objectidentifier module 402 that the factory machine 114 is also a drill.Consequently, the training module 402 associates the virtual contentassociated with the specific type of machine with the identification ofthe factory machine 114 based on their common type of machine. Othercombinations or permutations of different associations between virtualcontent and machines may be performed based on machine brand, model,location, usage, technology or other attributes or parameters.

In another example embodiment, the training module 404 generates alibrary of predefined conditions and corresponding virtual content. Thepredefined conditions may include viewing device predefined conditionsrelated to the viewing device 101 and physical object predefinedconditions related to the factory machine 114. The viewing devicepredefined conditions may be based on a state of the viewing device 101at the time of recording during operation of the virtual contentcreation module 214. Examples of viewing device predefined conditionsmay include a geographic location of the viewing device 101 at the timeof recording, a distance or range of distances between the viewingdevice 101 and the factory machine 114 at the time of recording, aposition and orientation of the viewing device 101 as determined bysensors 202 in the viewing device 101 at the time of recording. Otherexamples of viewing device predefined conditions may include a state ofthe wearer of the viewing device 101. Example of states of the wearermay be a calm user state, an agitated user state, or a nervous userstate. The state of the wearer may be identified using sensors (e.g.,heart monitor) connected to the wearer. The wearer of the viewing device101 may also select the state of the wearer to be included in theviewing device predefined conditions. Examples of physical objectpredefined conditions may include an operating status of the physicalobject (e.g., normal status, defective status) and a geographic locationof the physical object (e.g., GPS coordinates). The physical objectpredefined conditions may also be based on a state of the physicalobject (e.g., the operating status and geographic location of thephysical object) at the time of recording during operation of thevirtual content creation module 214.

Referring back to FIG. 2 , the virtual content access module 218generates a visualization of the virtual content based on a combinationof feature points of the physical object being viewed by the viewingdevice 101, and tracking data from sensors 202 internal to the viewingdevice 101 and tracking sensors 112 external to the viewing device 101.The virtual content access module 218 may be used to retrieve virtualcontent that was originally created on the viewing device 101 or onanother viewing device. FIG. 5 is a block diagram illustrating examplemodules of the virtual content access module 218. The virtual contentaccess module 218 is shown by way of example to include conditionsmodule 502 and a virtual content rendering module 504.

The conditions module 502 detects current conditions associated with theviewing device 101 and the factory machine 114. Examples of currentconditions include an identification of a physical object being viewedby the viewing device 101 and data tracking related to the viewingdevice 101 while viewing the physical object. The conditions module 502may use the physical object identifier module 402 to detect theidentification of the physical object presently being viewed the viewingdevice 101. The conditions module 502 may use the sensors 202 of theviewing device 101 and tracking sensors 112 external to the viewingdevice 101 to determine a present state (e.g., geographic location,position, orientation, and movement) of the viewing device 101.Furthermore, the conditions module 502 may communicate with the server110 or directly with the factory machine 114 to determine a currentoperating status (e.g., normal operation, defective, overheated,specific error code) of the factory machine 114. Therefore, the currentconditions include a combination of information based on theidentification of the factory machine 114, the present state of theviewing device 101, and the current operating status of the factorymachine 114.

The conditions module 502 may further retrieve predefined conditionsfrom a library of predefined conditions stored in the storage device 208or in the server 110. As previously described with respect to FIG. 4 ,the virtual content training module 216 stores predefined conditions andcorresponding virtual content in the library of predefined conditions inthe storage device 208. The library of predefined conditions andcorresponding virtual content or portions of the library can also beuploaded to the server 110 or to another viewing device. The conditionsmodule 502 accesses the library of predefined conditions and comparesthe current conditions associated with the viewing device 101 and thefactory machine 114 with the predefined conditions in the library ofpredefined conditions in the storage device 208. In one exampleembodiment, the conditions module 502 accesses predefined conditionsbased in part on a combination of an identification of the factorymachine 114, a geographic location of the viewing device 101, a presentstate of the viewing device 101, a current operating status of thefactory machine 114. For example, a predefined condition C includes anidentification of the factory machine 114 (e.g., machine M), apredefined distance range between the viewing device 101 and the factorymachine 114 (e.g., 2 to 3 feet), geographic location information of theviewing device 101 (e.g., GPS coordinates), an error operating status ofthe factory machine 114 (e.g., malfunction X). Virtual content Vincludes a video footage or animation of virtual objects showing how torepair malfunction X is associated with the predefined condition C. Theconditions module 502 compares the predefined condition C with thecurrent conditions associated with the viewing device 101 and thefactory machine 114. The conditions module 502 may be configured toretrieve the virtual content associated with the predefined condition Cin response to a combination of the current conditions associated withthe viewing device 101 and the factory machine 114 matching thepredefined conditions. For example, the conditions module 502 may beconfigured to retrieve the virtual content when all of the currentconditions associated with the viewing device 101 and the factorymachine 114 match the predefined conditions. Therefore, the conditionsmodule 502 retrieves the virtual content V only when the viewing device101 is viewing machine M at a distance of 2 feet of the machine M, andmachine M exhibits malfunction X. In another example, the conditionsmodule 502 may be configured to retrieve the virtual content when atleast one of the current conditions associated with the viewing device101 and the factory machine 114 match the predefined conditions.Therefore, the conditions module 502 retrieves the virtual content Vwhen the viewing device 101 is viewing machine Mat a distance of 5 feetof the machine M, and machine M exhibits a normal operating status. Inyet another example, the conditions module 502 retrieves all virtualcontent associated with the machine M regardless of the operating statusof the machine M.

Once the conditions module 502 retrieves the virtual content, theconditions module 502 provides the virtual content to the virtualcontent rendering module 504. In one example embodiment, the conditionsmodule 502 refers the virtual content rendering module 504 to therelevant virtual content locally stored in the storage device 208. Inanother example embodiment, the conditions module 502 refers the virtualrendering module 504 to the relevant virtual content remotely stored inthe server 110.

The virtual content rendering module 504 generates a visualization ofthe virtual content in the display 204. In one example embodiment, thevirtual content rendering module 504 renders an animatedthree-dimensional object based on a three-dimensional model or displaysa video footage, or a combination thereof. The virtual content renderingmodule 504 renders the virtual content to be displayed in relation to aline of sight between the eyes of the wearer of the viewing device 101and the physical object being viewed by the wearer of the viewing device101 through the display 204 of the viewing device 101. In anotherexample embodiment, the virtual content rendering module 504 displaysthe virtual content in the display 204 based on a position of thedisplay 204 of the viewing device 101 in relation to the physical objectbeing viewed by the viewing device 101. For example, if the display 204is oriented towards the left side of the physical object when the wearerof the viewing device 101 looks at the left side of the physical object,the virtual content rendering module 504 renders the virtual content onthe right side of the display 204 to compensate keep the virtual contentwithin the display 204.

In one example embodiment, the virtual content rendering module 504accesses a video file from the virtual content referred to by theconditions module 502. The virtual content rendering module 504generates a video footage in the display 204. The size of the videofootage may be adjusted and reduced to a portion of the display 204 soas not to entirely block a view of the wearer of the viewing device 101.For example, the video footage may be displayed in an upper corner ofthe display 204. In another example embodiment, the video footage may bedisplayed substantially over the entire surface of the display 204. Thevirtual content rendering module 504 may play the video file and pauseplayback in response to a command from a user interface (e.g., user tapsat a specific location on the viewing device 101 or says “pause” or“stop”). The virtual content rendering module 504 may pause playback ofthe video file when the wearer of the viewing device 101 turns away fromviewing the physical object. The virtual content rendering module 504may resume playback of the video file when the wearer of the viewingdevice 101 turns back to view the physical object after a presetduration.

In another example embodiment, the virtual content rendering module 504accesses a three-dimensional model of a virtual object from the virtualcontent referred to by the conditions module 502. For example, thethree-dimensional model may be a virtual arrow or specific effects(e.g., sparks) to guide the user of the viewing device 101 on how tooperate or repair the physical object. The three-dimensional model maybe assigned to a specific feature point (e.g., upper left corner of thephysical object). Consequently, the three-dimensional model of a virtualobject “follows” the specific feature point. For example, the virtualcontent rendering module 504 renders a virtual arrow that floats above aswitch. The virtual arrow can be animated to further gain the user'sattention on toggling the switch in a suggested direction. The virtualarrow is displayed next to the switch regardless of the position andorientation of the viewing device 101.

Each animation or virtual objects from the virtual content may betriggered based on the state of the physical object. For example, thevirtual content rendering module 504 may display a first virtual arrowabove a first switch of the physical object. After the user flips thefirst switch in the direction indicated by the first arrow, the firstvirtual arrow disappears from the display 204 and the virtual contentrendering module 504 displays a second virtual arrow above a lever ofthe physical object. Consequently, the virtual content rendering module504 displays portions of the virtual content in the display 204 in asequence based the user's manipulation on the physical object.

The shape, size, color, or appearance of the virtual content may changebased on the position and orientation of the viewing device 101 relativeto the physical object being viewed. For example, the virtual contentmay appear bigger as the viewing device 101 moves closer to the physicalobject associated with the virtual content. Similarly, the color of thevirtual content may change based on a current state of the physicalobject (e.g., red for malfunction, green for normal operation).

The virtual content generating module 504 may include a local renderingengine that generates a visualization of a three-dimensional virtualobject overlaid (e.g., superimposed upon) in a display in a line ofsight of the physical object. Cameras in sensors 202 can determine theline of sight of the viewing device 101. Gyroscope, compass, IMUS may beused to determine a position and orientation of the viewing device 101,and therefore a line of sight from the viewing device 101. The line ofsight may extend perpendicular to the display 204 of the viewing device101. The virtual content generating module 504 therefore manipulates aview of the three-dimensional virtual object based on a position of thephysical object relative to the camera of the viewing device 101. Forexample, the virtual content generating module 504 displays thethree-dimensional virtual object from different angles based on positionof the camera of the viewing device 101 moving around the physicalobject.

Referring back to FIG. 2 , the storage device 208 may be configured tostore a library or a database of virtual content, correspondingpredefined conditions including predefined conditions of viewing devicesand predefined conditions of physical objects. In another exampleembodiment, the database may also include physical object identifierssuch as visual references. Examples of visual references include amachine-readable code on a factory machine M. The machine-readable codemay correspond to a three-dimensional virtual model of the factorymachine M that can be viewed from different angles by manipulating theposition of the viewing device 101 relative to the machine-readablecode.

In one example embodiment, the storage device 208 includes a primarycontent dataset and a contextual content dataset. The primary contentdataset includes, for example, a first set of identifying feature pointsand corresponding virtual content. For example, identifying featurepoints may be associated with one or more virtual object models. Theprimary content dataset may include a core set of identifying featurepoints or the most popular identifying feature points determined by theviewing device 101 or by the server 110. The core set of identifyingfeature points may include a limited number of identifying featurepoints identified by the viewing device 101 or by the server 110. Forexample, the core set of identifying feature points may include theidentifying feature points depicting the most viewed or used machinesand their corresponding virtual content. In another example, the server110 may generate the first set of identifying feature points based onthe most popular or often scanned identifying feature points (e.g., ofmost often viewed machines) received at the server 110. In yet anotherexample, the primary content dataset includes the first set ofidentifying feature points and corresponding virtual content frommachines that often have the most malfunctions, from machines that arebased in at a location (e.g., machines from factory region R), frommachines of a same manufacturer, from machines from which a user of theviewing device 101 has exclusive access, from machines most oftenoperated by the user of the viewing device 101, or from machinesassigned to the user of the viewing device 101 based on a job or titledescription.

The contextual content dataset includes, for example, a second set ofidentifying feature points and corresponding virtual content retrievedfrom the server 110. For example, identifying feature points capturedwith the viewing device 101 that are not locally recognized in theprimary content dataset are submitted to the server 110 for recognition.If the captured identifying feature points are recognized by the server110, the corresponding virtual content may be downloaded at the viewingdevice 101 and stored in the contextual content dataset of the viewingdevice 101. Thus, the contextual content dataset relies on the contextin which the viewing device 101 is being used, unlike the primarycontent dataset which may be preloaded on viewing device 101. As such,the contextual content dataset depends on objects or identifying featurepoints scanned by the physical object identifier module 402 of theviewing device 101.

In one example embodiment, the viewing device 101 may communicate overthe network 108 with the server 110 to retrieve a portion of a databaseof visual references, corresponding three-dimensional virtual objects,and corresponding interactive features of the three-dimensional virtualobjects. The network 108 may be any network that enables communicationbetween or among machines, databases, and devices (e.g., the viewingdevice 101).

Any one or more of the modules described herein may be implemented usinghardware (e.g., a processor of a machine) or a combination of hardwareand software. For example, any module described herein may configure aprocessor to perform the operations described herein for that module.Moreover, any two or more of these modules may be combined into a singlemodule, and the functions described herein for a single module may besubdivided among multiple modules. Furthermore, according to variousexample embodiments, modules described herein as being implementedwithin a single machine, database, or device may be distributed acrossmultiple machines, databases, or devices.

FIG. 6 is a block diagram illustrating example modules (e.g.,components) executing on one or more processors of the server 110. Theserver 110 is shown by way of example to include a server AR application602 and a database 604. The server AR application 602 may receivevirtual content from the viewing device 101 and other viewing devices.In another example embodiment, the server AR application 602 may providevirtual content to the viewing device 101 and other viewing devices. Theserver AR application 602 is shown by way of example to include anauthentication module 606, a virtual content receiver module 608, and avirtual content access module 610.

The authentication module 606 authenticates viewing devices attemptingto upload virtual content to the server 110 or download virtual contentfrom the server 110. For example, the authentication module 606 usesbiometric information from users of the respective viewing devices. Theauthentication module 606 receives biometric information from a viewingdevice and compares the received biometric information with a library ofbiometric information. Once the authentication module 606 matches thebiometric information with the biometric information from the library,the user associated with the corresponding biometric information isauthenticated and can upload or download virtual content to and from thedatabase 604 of the server 110. In one example embodiment, users mayhave different types of access privileges based on their access status.For example, an admin user may be granted full access to all virtualcontent in the database 604. Some users may have limited uploadprivilege while others may have limited download privilege. In otherexample embodiments, a user may have download privilege to virtualcontent relevant to their job status. For example, a technician workingon a drill machine may have access to all virtual content related todrill machines. Similarly, a user may have limited upload privilege forvirtual content relevant to a job status. For example, a technicianhaving access privilege to machines A, B, and C may upload virtualcontent only with respect to those machines A, B, and C.

Once the authentication module 606 authenticates the user of the viewingdevice 101, the server AR application 602 proceeds with receivingvirtual content from the authenticated viewing device with the virtualcontent receiver module 608 and providing access to virtual contentstored in the database 604 with virtual content access module 610.

The virtual content receiver module 608 receives virtual contentauthored at the viewing device 101. For example, the virtual contentreceiver module 608 receives, from the viewing device 101, virtualcontent associated with a physical object and predefined conditionsrelated to the physical object and the viewing device 101. As previouslydescribed, the virtual content may include a three-dimensional model ofa virtual object, a video clip, or any other multimedia item. In oneexample embodiment, the virtual content receiver module 608 may receive,in addition to the virtual content, an identification of a physicalobject, a geographic location of the physical object, a location of theviewing device, viewing device predefined conditions, and physicalobject predefined conditions.

The viewing device predefined conditions may include conditions relatedto the viewing device associated with the virtual content. Thus, thevirtual content may be triggered when current conditions of the viewingdevice match the viewing device predefined conditions. Examples ofviewing device predefined conditions include a time and location ofviewing, a position and orientation of the viewing device relative tothe physical object. The viewing device predefined conditions mayinclude conditions of the viewing device at the time of recording theuser interacting (e.g., operating or fixing) with the physical object.

The physical object predefined conditions may include conditions relatedto the physical object associated with the virtual content. Thus, thevirtual content may be triggered when current conditions of the physicalobject match the physical object predefined conditions. Examples ofphysical object predefined conditions include a geographic location ofthe physical object (e.g., factory F), an operating status of thephysical object (e.g., malfunction M), last maintenance operation (e.g.,time T), user who performed last operation on the physical object (e.g.,user U). The viewing device predefined conditions may include conditionsof the physical object at the time of recording the user interacting(e.g., operating or fixing) with the physical object.

The viewing device 101 may generate metadata comprising theidentification of the physical object, the viewing device predefinedconditions, the physical object predefined conditions and associate themetadata with the virtual content authored at the viewing device 101.The viewing device 101 uploads the virtual content (e.g., video clip oranimation of virtual object) and the corresponding metadata (e.g.,identification of physical object related to the video clip, viewingdevice predefined conditions, physical object predefined conditions) tothe virtual content receiver module 608. The virtual content receivermodule 608 stores the virtual content and metadata in the database 604.

In one example embodiment, the database 604 includes a content dataset612 and a virtual content dataset 614. The content dataset 612 stores,for example, an identification of the physical object (e.g., featurespoints of the physical object) and corresponding metadata (e.g.,conditions related to when the virtual content is to be rendered in theviewing device) that are associated with the identification of thephysical object. As previously described, the metadata may includeviewing device predefined conditions, physical object predefinedconditions, or a combination thereof. In this example, the contentdataset 604 may include a library of identifications of physicalobjects, a library of viewing device predefined conditions, and alibrary of physical object predefined conditions previously receivedfrom the viewing device 101, other viewing devices, and from a clientdevice of the server 110. For example, a client device may have uploadedan initial library of identifications of physical objects, andcorresponding metadata (e.g., viewing device predefined conditions, andphysical object predefined conditions) to form an initial contentdataset 612 in the database 604.

In other example embodiments, the content dataset 612 may be divided orotherwise organized based on geographic locations of the physicalobjects. For example, the content dataset 612 may include a subdatasetfor physical objects located at a particular manufacturing facility. Thecontent dataset 612 may also be organized by privilege access. Forexample, a senior technician may have access to a larger number of itemsfrom the content dataset 612 than a junior technician.

The virtual content dataset 614 stores the virtual content (e.g.,three-dimensional model of a virtual object, media clip) received fromthe viewing device 101 or other viewing devices. The virtual contentdataset 614 stores, for example, virtual content that are associatedwith the identification of the physical object, current conditions ofthe viewing device or physical object. In this example, the contentdataset 604 may include a library of virtual content (e.g.,three-dimensional models of virtual objects) previously received fromthe viewing device 101, other viewing devices, and from a client deviceof the server 110. For example, the client device may have uploaded aninitial library of virtual content to form an initial virtual contentdataset 614 in the database 604.

The virtual content access module 610 provides virtual content from thevirtual content data 614 to the viewing device MI or other authenticatedviewing devices. The virtual content access module 610 may receive anotification from the viewing device 101 of an identification of aphysical object (e.g., using feature points of the physical object). Thevirtual content access module 610 retrieves the corresponding virtualcontent associated with the identified physical object and provides thevirtual content to the viewing device 101. In another exampleembodiment, the virtual content access module 610 receives currentconditions related to the viewing device 101 and the physical object.For example, the virtual content access module 610 receives a locationof the viewing device 101 (e.g., factory F), an operating state of thephysical object (e.g., malfunction M), a relative position of theviewing device 101 to the physical object (e.g., viewing device lookingat a left side of the physical object), and retrieves the correspondingvirtual content based on the received current conditions of the viewingdevice 101 and the physical object. The virtual content access module610 sends the corresponding virtual content to the viewing device 101.

FIG. 7 is an interaction diagram illustrating interactions between aviewing device for creating virtual content and a server. Theinteractions may be performed by the viewing device 101 and the server110 and, accordingly, are described merely by way of example withreference thereto.

At operation 702, the viewing device 101 views a physical object,identifies identifiers of the physical object, and conditions related tothe viewing device and the physical object. For example, the viewingdevice 101 is used to view the factory machine 114. The viewing device101 may be five feet away from the factory machine 114. The factorymachine 114 has a malfunction status. The viewing device 101 identifiesfeature points (e.g., QR code on the machine), determines that thefactory machine 114 is located five feet in front of the viewing device101, determines that the viewing device 101 is pointed to the front ofthe factory machine 114, and determines a malfunction status based on anerror output displayed on the factory machine 114 (or based on acommunication from the factory machine 114 to the viewing device 101 orto the server 110). In response to a command from the user, the viewingdevice 101 records the user operating on the physical object (seeoperation 704). For example, the viewing device 101 records the userfixing the factory machine 114 to render the factory machineoperational. The viewing device 101 generates virtual content based onthe user operating the physical object as shown in operation 706. Forexample, the viewing device 101 generates a video footage or ananimation of a three-dimensional object based on the point of view ofthe viewing device 101 at the time the user of the viewing device 101was fixing the factory machine 114. The viewing device 101 associatesthe virtual content with the identification of the physical object, theconditions of the viewing device 101, and the conditions of the physicalobject. For example, the viewing device 101 associates thethree-dimensional object with feature points of the factory machine 114,the location and orientation of the viewing device 101 relative to thefactory machine 114, and the error output displayed on the factorymachine 114. The conditions of the viewing device 101 and the conditionsof the physical object at the time the user operated on the physicalobject may be defined as predefined conditions of the viewing device andphysical object. Operation 706 may be implemented using the virtualcontent creation module 214 and virtual content training module 216.

The viewing device 101 stores the physical object identifiers,predefined conditions of the viewing device 101, predefined conditionsof the physical object, and authored virtual content in a local storage(e.g., storage device 208) of the viewing device 101. In another exampleembodiment, the viewing device 101 communicates the physical objectidentifiers, predefined conditions of the viewing device 101, predefinedconditions of the physical object, and authored virtual content to theserver 110 (see operation 708). In one example implementation, theviewing device 101 communicates with the virtual content receiver module608 of the server 110. The server 110 stores the physical objectidentifiers, predefined conditions of the viewing device 101, predefinedconditions of the physical object, and authored virtual content in thedatabase 604 as shown in operation 710. For example, the server 110 maystore the physical object identifiers, predefined conditions of theviewing device 101, and predefined conditions of the physical object inthe content dataset 612 in the database 604. The corresponding authoredvirtual content may be stored in the virtual content dataset 614 in thedatabase 604.

At a later time, the viewing device 101 may be used to view the samephysical object (e.g., factory machine 114) at operation 712. Theviewing device 101 identifies identifiers of the physical object, andconditions related to the viewing device and the physical object at thatlater viewing time. For example, the viewing device 101 identifiesfeature points of the factory machine 114, determines the relativeposition and orientation of the viewing device 101 with respect to thefactory machine 114, and determines an operation status based on anoutput displayed on the factory machine 114. These conditions at thelater time may be referred to as current conditions. Operation 712 maybe implemented with the conditions module 502 from the virtual contentaccess module 218 of the viewing device 101.

Subsequently, at operation 714, the viewing device 101 retrieves, from alocal memory (e.g., storage device 208), the virtual content associatedwith identifiers of the physical object, conditions related to theviewing device at that later viewing time, conditions related to thephysical object at that later viewing time, or a combination thereof.For example, the viewing device 101 retrieves a three-dimensional modelof a virtual object (e.g., virtual parts) associated with the factorymachine 114. In another example, the viewing device 101 retrieves thethree-dimensional model of the virtual object when current conditions ofthe viewing device 101 match the predefined conditions of the viewingdevice 101. For example, the viewing device 101 retrieves thethree-dimensional model of the virtual object when the viewing device101 detects that it is viewing the factory machine 114 located five feetaway from the viewing device 101. In this example, the three-dimensionalmodel of the virtual object may be retrieved regardless of an operatingstatus of the factory machine 114. In another example, the viewingdevice 101 retrieves the three-dimensional model of the virtual object(e.g., animated parts or arrow) when the viewing device 101 detects thatit is viewing the factory machine 114 located five feet away from theviewing device 101 and that the factory machine 114 has a malfunction.The three-dimensional model of the factory machine 114 may be associatedwith the specific malfunction of the factory machine 114.

Once the viewing device 101 retrieves the virtual content, the viewingdevice 101 renders the virtual content in the display 204 as shown inoperation 716. For example, the viewing device may generate a videoclip, shown from the same point of view of the viewing device 101,showing how to fix the factory machine 114, or render athree-dimensional virtual objects showing which parts are needed to fixthe factory machine 114. The viewing device 101 may generate anotification to the user of the viewing device 101 that virtual contentis available to be displayed. The notification may further identify thatthe virtual content is based on the identification of the physicalobject, current conditions of the viewing device 101, or currentconditions of the physical object. For example, the notification mayshow: “a video clip showing how to operate the drill machine in front ofyou is available for viewing, say ‘play it’ to view the video clip”, or“machine in front of you is malfunctioning, content showing how torepair is available for viewing, say ‘show me how’ to view the content.”In response to a vocal command from the user, the viewing device 101displays the corresponding virtual content in the display 204. Operation716 may be implemented with the virtual content rendering module 504 ofthe virtual content access module 218 of the viewing device 101.

FIG. 8 is an interaction diagram illustrating interactions between aviewing device for accessing virtual content and a server. Theinteractions may be performed by the viewing device 101 and the server110 and, accordingly, are described merely by way of example withreference thereto.

At operation 802, the server 110 receives physical object identifiers,predefined conditions of the viewing device and the physical object, andcorresponding authored virtual content from the viewing device 101 orother viewing devices. The server 110 stores the physical objectidentifiers, predefined conditions of the viewing device and thephysical object, and corresponding authored virtual content in thedatabase 604 of the server 110 (as shown in operation 804). For example,the physical object identifiers, predefined conditions of the viewingdevice and the physical object may be stored in the content dataset 612of the database 604. The corresponding authored virtual content may bestored in the virtual content dataset 614 of the database 614.

Subsequently, the viewing device 101 may be used to view a physicalobject (e.g., factory machine 114). At operation 806, the viewing device101 identifies identifiers of the physical object, and currentconditions related to the viewing device and the physical object.Current conditions may refer to conditions (of the viewing device 101and the physical object) at the time the viewing device 101 views thephysical object. For example, the viewing device 101 identifies featurepoints of the factory machine 114, determines the relative position andorientation of the viewing device 101 with respect to the factorymachine 114, and determines a current operation status based on anoutput displayed on the factory machine 114 at the time the viewingdevice 101 views the physical object. Operation 806 may be implementedwith the physical object identifier module 402 and the conditions module502 of the viewing device 101.

The viewing device 101 determines whether the identifiers of thephysical object and the current conditions match with any identifiersand predefined conditions in the local memory (e.g., local storagedevice 208). If the viewing device 101 determines that the local memorydoes not include matching identifiers or predefined conditions, theviewing device 101 communicates the identifiers of the physical objectand the current conditions to the server 110 (as shown in operation810). In one example implementation, the viewing device 101 communicateswith the virtual content access module 610 from the server 110 torequest for the virtual content associated with the identifiers of thephysical object and the current conditions

Subsequently, the server 110 retrieves, from the database 604, thevirtual content associated with identifiers of the physical object,current conditions related to the viewing device, conditions related tothe physical object, or a combination thereof at operation 812. Forexample, the server 110 retrieves a three-dimensional model of a virtualobject (e.g., virtual parts of the physical object) associated with thephysical object. In another example, the server 110 retrieves thethree-dimensional model of the virtual object when current conditions ofthe viewing device 101 match the predefined conditions of the viewingdevice 101 stored in the database 604. For example, the server 110retrieves the three-dimensional model of the virtual object when thecurrent conditions indicate that the viewing device 101 is located fivefeet away from the physical object. In this example, thethree-dimensional model of the virtual object may be retrievedregardless of an operating status of the physical object. In anotherexample, the server 110 retrieves the three-dimensional model of thevirtual object (e.g., animated parts or arrow) when the viewing device101 detects that it is viewing the physical object located five feetaway from the viewing device 101 and that the physical object has aparticular malfunction.

Once the server 110 retrieves the virtual content associated with thephysical object identifiers or current conditions of the viewing device101 or of the physical object, the server 110 communicates the virtualcontent to the viewing device 101 as shown in operation 814.Subsequently, the viewing device 101 renders the virtual content in thedisplay 204 (operation 816). The viewing device 101 may generate thevirtual content in a line of sight of the wearer of the viewing device101 such that the virtual content is perceived as interacting with thephysical object. For example, a virtual arrow may be displayed above acorresponding physical component (e.g., an electrical switch). The size,shape, and color of the virtual object may be adjusted based on theposition and orientation of the viewing device 101 relative to thephysical object. For example, the virtual object may appear larger whenthe distance between the viewing device and the physical objectdecreases. Operation 816 may be implemented with the virtual contentrendering module 504 of the viewing device 101.

FIG. 9 is a flowchart illustrating an example operation of authoringvirtual content from a viewing device (such as viewing device 101) foran augmented reality application. At operation 902, the viewing device101 is used to view a physical object (such as factory machine 114) anddetermine physical object identifiers and current conditions associatedwith the physical object and the viewing device 101. In one exampleembodiment, sensors 202 in the viewing device 101 may be used todetermine a current condition associated with the viewing device 101while tracking sensors 112 may be used to determine a current conditionassociated with the factory machine 114. Sensors 202 may also be used todetermine a current condition of the factory machine 114 (e.g.,detecting flashing red lights from the factory machine 114 or reading anerror message on a display of the factory machine 114). Factory machine114 may also wireless communicate a status message to the server 110 orto the viewing device 101.

At operation 904, the viewing device IOI records the user operating thephysical object and conditions changes to the physical object and theviewing device (e.g., position, temperature, orientation). For example,the viewing device 101 may record a video footage of the user fixing thephysical object. In another example, the viewing device 101 trackschanges to feature points on the physical object to record changes tothe physical object.

The viewing device 101 generates a virtual content based on the useroperating the physical object and changes in conditions to the physicalobject and the viewing device as shown in operation 906. For example,the virtual content may include the video clip and is associated with amalfunction status of the physical object. The changes in conditionsshow that the physical object was initially defective and is back tooperational after the user has operated on the physical object. Theviewing device 101 associates the virtual content with the physicalobject identifiers and predefined conditions. Predefined conditions mayinclude the initial conditions of the physical object or viewing deviceprior to the user fixing the physical object. In another exampleembodiment, the predefined conditions may selected by user of theviewing device 101. For example, the user may specify that the virtualcontent is to be triggered for specific conditions (e.g., during nighttime operations between 9 pm and 6 am). Operation 906 may beimplemented, for example, using the virtual content creation module 214.

Subsequently, the viewing device 101 associates the authored virtualcontent with the physical object identifiers and predefined conditions(of the viewing device or physical object) at operation 908. In oneexample embodiment, operation 908 may be implemented using the trainingmodule 404.

At operation 910, the viewing device 101 stores the virtual content andcorresponding predefined conditions in a local memory (e.g., localstorage device 208) or in a remote storage device (e.g., database 604).In one example embodiment, operation 910 may be implemented using thetraining module 404.

FIG. 10 is a flowchart illustrating an example operation of retrieving avirtual content at a viewing device (e.g., viewing device 101). Atoperation 1002, the viewing device 101 identifies physical objectidentifiers and current conditions (related to a physical object or theviewing device 101), and optionally from other viewing devices. Forexample, the viewing device 101 may determine that another viewingdevice is experiencing high heat (e.g., temperature above a presetthreshold) in a same floor of a factory. Operation 1002 may beimplemented using the physical object identifier module 402 and theconditions module 502.

At operation 1004, the viewing device 101 retrieves the correspondingvirtual content associated with the physical object and the currentconditions of the physical object and the viewing device. For example,the viewing device 101 retrieves evacuation directions in the form ofvirtual arrows on the floor in response to the other viewing devicesexperiencing high heat on the same floor of the factory. The viewingdevice 101 then renders the virtual arrows in the display 204 of theviewing device 101 to show an evacuation route. The retrieval of thevirtual content may be implemented at the viewing device 101 by thevirtual content access module 218. The display of the virtual arrows maybe implemented at the viewing device 101 by the virtual contentrendering module 504.

FIG. 11 is a diagram illustrating an example operation for generatingvirtual content related to a physical object for an augmented realityapplication at a mobile device. The viewing device 101 may include ahead mounted device having a camera 1102 and a display 1104. The viewingdevice 101 may be pointed at a physical object 1110. The camera 1102captures an image of the physical object 1110 and displays virtualcontent 1108 related to the physical object 1110 (and related to currentconditions of the viewing device 101 and/or the physical object 1110) inthe transparent display 1104. Identifiers and current conditions relatedto the physical object 1110 are determined by the viewing device 101based on a combination of the image of the physical object 1110, ambientconditions of the viewing device 101 (e.g., a location of the viewingdevice, an ambient temperature of a room where the viewing device islocated, an authentication of the user of the viewing device, and otherpredefined conditions). The user of the viewing device 101 may authorvirtual content based on a selection of content templates or may createvirtual content (e.g., audio/video media, three dimensional models) andassociate a selected virtual content to the physical object 1110 and/orcurrent conditions. The selection of the three dimensional model may beperformed via a variety of user interfaces (e.g., voice recognition,visual gestures—moving fingers in a predefined geometric pattern). Theuser thus can specify which virtual content is associated with thephysical object 1110 and where to display the selected virtual content1108 in relation to a view of the physical object 1110 through thedisplay 1104. In another embodiment, the association of the selectedvirtual content 1108 with the physical object 1110 may be stored at theserver 110.

FIG. 14 is a diagram illustrating a front view of a head mounted device1400. FIG. 15 is a block diagram illustrating a side view of a headmounted device 1400, according to some example embodiments. The HMD 1400includes a helmet 1402 connected to a visor 1404. The helmet 1402 mayinclude sensors (e.g., optical and audio sensors) 1408 and 1410 disposedin the front, back, and a top section 1406 of the helmet 1402. Displaylenses 1412 are mounted on a lens frame 1414. The lens frame 1414 isconnected to an underside of the helmet 1402. The display lenses 1412include a partially transparent display. The helmet 1402 includes an eyegaze tracking sensor 1411 in a housing of the helmet 1402. The eye gazetracking sensor 1411 tracks a position of the pupil of the eye of thewearer of the helmet 1402 to determine a direction at which the user isstarting. The eye gaze tracking sensor 1411 also determines how long theuser is staring at a particular point. The eye gaze tracking sensor 1411also determines whether the user is blinking and how many times the useris blinking, time duration of each blink. Programmable operations of theHMD 1400 may be defined based on the staring duration, the number of eyeblinks, a head movement, or a combination thereof. Data from the eyegaze tracking sensor 1411 may be combined with data from other sensorssuch Inertial Measurement Unit (IMU), gyroscope data, temperature data,and location data, to generate a range of conditions associated with thevirtual content created at the helmet 1402. For example, the helmet 1402may be used to record a video of a user of the helmet 1402 fixing amachine. When another helmet user senses that a similar machine, viewedby the other helmet, is broken, the virtual content (video clip) isdisplayed in that helmet. For example, the video clip is displayed whenthe helmet is in a position/orientation similar to theposition/orientation when the helmet 1402 was used to record the videoclip.

Modules, Components and Logic

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A hardware module is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain manner. In example embodiments, oneor more computer systems (e.g., a standalone, client, or server computersystem) or one or more hardware modules of a computer system (e.g., aprocessor or a group of processors) may be configured by software (e.g.,an application or application portion) as a hardware module thatoperates to perform certain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired) or temporarilyconfigured (e.g., programmed) to operate in a certain manner and/or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connect the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein maybeperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or asa“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork and via one or more appropriate interfaces (e.g., APIs).

Electronic Apparatus and System

Example embodiments may be implemented in digital electronic circuitry,or in computer hardware, firmware, software, or in combinations of them.Example embodiments may be implemented using a computer program product,e.g., a computer program tangibly embodied in an information carrier,e.g., in a machine-readable medium for execution by, or to control theoperation of, data processing apparatus, e.g., a programmable processor,a computer, or multiple computers.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, subroutine,or other unit suitable for use in a computing environment. A computerprogram can be deployed to be executed on one computer or on multiplecomputers at one site or distributed across multiple sites andinterconnected by a communication network.

In example embodiments, operations may be performed by one or moreprogrammable processors executing a computer program to performfunctions by operating on input data and generating output. Methodoperations can also be performed by, and apparatus of exampleembodiments may be implemented as, special purpose logic circuitry(e.g., a FPGA or an ASIC).

A computing system can include clients and servers. A client and serverare generally remote from each other and typically interact through acommunication network. The relationship of client and server arises byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other. In embodimentsdeploying a programmable computing system, it will be appreciated thatboth hardware and software architectures merit consideration.Specifically, it will be appreciated that the choice of whether toimplement certain functionality in permanently configured hardware(e.g., an ASIC), in temporarily configured hardware (e.g., a combinationof software and a programmable processor), or a combination ofpermanently and temporarily configured hardware may be a design choice.Below are set out hardware (e.g., machine) and software architecturesthat may be deployed, in various example embodiments.

Example Machine Architecture and Machine-Readable Medium

FIG. 12 is a block diagram of a machine in the example form of acomputer system 1200 within which instructions 1224 for causing themachine to perform any one or more of the methodologies discussed hereinmay be executed. In alternative embodiments, the machine operates as astandalone device or may be connected (e.g., networked) to othermachines. In a networked deployment, the machine may operate in thecapacity of a server or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine may be a personal computer (PC), atablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), acellular telephone, a web appliance, a network router, switch or bridge,or any machine capable of executing instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term“machine” shall alsobe taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 1200 includes a processor 1202 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 1204 and a static memory 1206, which communicatewith each other via a bus 1208. The computer system 1200 may furtherinclude a video display unit 1210 (e.g., a liquid crystal display (LCD)or a cathode ray tube (CRT)). The computer system 1200 also includes analphanumeric input device 1212 (e.g., a keyboard), a user interface (UI)navigation (or cursor control) device 1214 (e.g., a mouse), a disk driveunit 1216, a signal generation device 1218 (e.g., a speaker) and anetwork interface device 1220.

Machine-Readable Medium

The disk drive unit 1216 includes a machine-readable medium 1222 onwhich is stored one or more sets of data structures and instructions1224 (e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1224 mayalso reside, completely or at least partially, within the main memory1204 and/or within the processor 1202 during execution thereof by thecomputer system 1200, the main memory 1204 and the processor 1202 alsoconstituting machine-readable media. The instructions 1224 may alsoreside, completely or at least partially, within the static memory 1206.

While the machine-readable medium 1222 is shown in an example embodimentto be a single medium, the term “machine-readable medium” may include asingle medium or multiple media (e.g., a centralized or distributeddatabase, and/or associated caches and servers) that store the one ormore instructions 1224 or data structures. The term “machine-readablemedium” shall also be taken to include any tangible medium that iscapable of storing, encoding or carrying instructions for execution bythe machine and that cause the machine to perform any one or more of themethodologies of the present embodiments, or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude non-volatile memory, including by way of example semiconductormemory devices (e.g., Erasable Programmable Read-Only Memory (EPROM),Electrically Erasable Programmable Read-Only Memory (EEPROM), and flashmemory devices); magnetic disks such as internal hard disks andremovable disks; magneto-optical disks; and compact disc-read-onlymemory (CD-ROM) and digital versatile disc (or digital video disc)read-only memory (DVD-ROM) disks.

Transmission Medium

The instructions 1224 may further be transmitted or received over acommunications network 1226 using a transmission medium. Theinstructions 1224 may be transmitted using the network interface device1220 and any one of a number of well-known transfer protocols (e.g.,HTTP). Examples of communication networks include a LAN, a WAN, theInternet, mobile telephone networks, POTS networks, and wireless datanetworks (e.g., Wi-Fi and WiMAX networks). The term “transmissionmedium” shall be taken to include any intangible medium capable ofstoring, encoding, or carrying instructions for execution by themachine, and includes digital or analog communications signals or otherintangible media to facilitate communication of such software.

Example Mobile Device

FIG. 13 is a block diagram illustrating a mobile device 1300, accordingto an example embodiment. The mobile device 1300 may include a processor1302. The processor 1302 may be any of a variety of different types ofcommercially available processors 1302 suitable for mobile devices 1300(for example, an XScale architecture microprocessor, a microprocessorwithout interlocked pipeline stages (MIPS) architecture processor, oranother type of processor 1302). A memory 1304, such as a random accessmemory (RAM), a flash memory, or other type of memory, is typicallyaccessible to the processor 1302. The memory 1304 may be adapted tostore an operating system (OS) 1306, as well as application programs1308, such as a mobile location enabled application that may providelocation based services to a user. The processor 1302 may be coupled,either directly or via appropriate intermediary hardware, to a display1310 and to one or more input/output (I/O) devices 1312, such as akeypad, a touch panel sensor, a microphone, and the like. Similarly, insome embodiments, the processor 1302 may be coupled to a transceiver1314 that interfaces with an antenna 1316. The transceiver 1314 may beconfigured to both transmit and receive cellular network signals,wireless data signals, or other types of signals via the antenna 1316,depending on the nature of the mobile device 1300. Further, in someconfigurations, a GPS receiver 1318 may also make use of the antenna1316 to receive GPS signals.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader scope of the present disclosure. Accordingly, the specificationand drawings are to be regarded in an illustrative rather than arestrictive sense. The accompanying drawings that form a part hereof,show by way of illustration, and not of limitation, specific embodimentsin which the subject matter may be practiced. The embodimentsillustrated are described in sufficient detail to enable those skilledin the art to practice the teachings disclosed herein. Other embodimentsmay be utilized and derived therefrom, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. This Detailed Description, therefore, is not to betaken in a limiting sense, and the scope of various embodiments isdefined only by the appended claims, along with the full range ofequivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in a single embodiment for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter lies in less than allfeatures of a single disclosed embodiment. Thus the following claims arehereby incorporated into the Detailed Description, with each claimstanding on its own as a separate embodiment.

What is claimed is:
 1. A computing device comprising: one or moreprocessors; and a non-transitory storage medium having instructionsencoded thereon that, when executed by the one or more processors, causethe one or more processors to: access a video that was captured by acamera, the video depicting a user performing a physical operation inrelation to a physical object, the physical object associated with anobject identifier; determine metadata identifying positions of thecamera relative to the physical object during the capturing of thevideo; identify feature points of the physical object in the video;retrieve a virtual object based on the feature points of the physicalobject; generate virtual content based on the video, the metadata, andthe virtual object; and store the object identifier and the virtualcontent relationally in a data storage that stores a plurality of setsof virtual content associated with a plurality of physical objects andcorresponding object identifiers.
 2. The computing device of claim 1,wherein an operation on the physical object is recorded by the camera inresponse to at least a verbal recording instruction from a user of thecamera to form the virtual content.
 3. The computing device of claim 1,wherein the metadata further comprise at least an orientation of thecamera relative to a physical object, an operational status of thephysical object, or a temperature range related to the physical object.4. The computing device of claim 1, wherein the instructions furthercause the one or more processors to: receive an image of the physicalobject from a client device; identify the object identifier based on theimage of the physical object; retrieve the virtual content based on theobject identifier; cause the virtual content to be displayed at adisplay of the client device.
 5. The computing device of claim 4,wherein the image of the physical object includes a quick response (QR)code, the QR code corresponding to the object identifier of the physicalobject.
 6. The computing device of claim 5, wherein the instructionsfurther cause the one or more processors to: extract the QR code fromthe image of the physical object; and identify the object identifierbased on the QR code.
 7. The computing device of claim 1, whereinphysical object is a machine of a particular type, the operation of thephysical operation in relation to the physical object includes anoperation of the machine, and the virtual content includes a videoillustrating an operation of the particular type of machine.
 8. Thecomputing device of claim 1, wherein physical object is a machine, andthe operation of the physical operation in relation to the physicalobject includes an operation of the machine.
 9. The computing device ofclaim 1, wherein the instructions further cause the one or moreprocessors to: determine a geographic location of the camera; access alibrary containing information describing geographic locations of aplurality of physical objects and respective object identifiers; anddetermine an object identifier based on the geographic location of thecamera and the library.
 10. A computer implemented method, the methodcomprising: accessing a video that was captured by a camera, the videodepicting a user performing a physical operation in relation to aphysical object, the physical object associated with an objectidentifier; determining metadata identifying positions of the camerarelative to the physical object during the capturing of the video;identifying feature points of the physical object in the video;retrieving a virtual object based on the feature points of the physicalobject; generating virtual content based on the video, the metadata, andthe virtual object; and storing the object identifier and the virtualcontent relationally in a data storage that stores a plurality of setsof virtual content associated with a plurality of physical objects andcorresponding object identifiers.
 11. The method of claim 10, wherein anoperation on the physical object is recorded by the camera in responseto at least a verbal recording instruction from a user of the camera toform the virtual content.
 12. The method of claim 10, wherein themetadata further comprise at least an orientation of the camera relativeto a physical object, an operational status of the physical object, or atemperature range related to the physical object.
 13. The method ofclaim 10, the method further comprising: receiving an image of thephysical object from a client device; identifying the object identifierbased on the image of the physical object; retrieving the virtualcontent based on the object identifier; causing the virtual content tobe displayed at a display of the client device.
 14. The method of claim13, wherein the image of the physical object includes a quick response(QR) code, the QR code corresponding to the object identifier of thephysical object.
 15. The method of claim 14, the method furthercomprising: extracting the QR code from the image of the physicalobject; and identifying the object identifier based on the QR code. 16.The method of claim 10, wherein physical object is a machine of aparticular type, the operation of the physical operation in relation tothe physical object includes an operation of the machine, and thevirtual content includes a video illustrating an operation of theparticular type of machine.
 17. The method of claim 10, wherein physicalobject is a machine, and the operation of the physical operation inrelation to the physical object includes an operation of the machine.18. The method of claim 10, the method further comprising: determining ageographic location of the camera; accessing a library containinginformation describing geographic locations of a plurality of physicalobjects and respective object identifiers; and determining an objectidentifier based on the geographic location of the camera and thelibrary.
 19. A computer program product, comprising a non-transitorystorage medium having instructions encoded thereon that, when executedby one or more processors, cause the one or more processors to: access avideo that was captured by a camera, the video depicting a userperforming a physical operation in relation to a physical object, thephysical object associated with an object identifier; determine metadataidentifying positions of the camera relative to the physical objectduring the capturing of the video; identify feature points of thephysical object in the video; retrieve a virtual object based on thefeature points of the physical object; generate virtual content based onthe video, the metadata, and the virtual object; and store the objectidentifier and the virtual content relationally in a data storage thatstores a plurality of sets of virtual content associated with aplurality of physical objects and corresponding object identifiers. 20.The computer program product of claim 19, wherein the instructionsfurther cause the one or more processors to: receive an image of thephysical object from a client device; identify the object identifierbased on the image of the physical object; retrieve the virtual contentbased on the object identifier; cause the virtual content to bedisplayed at a display of the client device.